Skip to content

ci(release): shard Linux integration, slim macOS, promote verified tags - #3771

Merged
Daniel Meppiel (danielmeppiel) merged 6 commits into
mainfrom
danielmeppiel-ci-release-speed-extract
Nov 17, 2026
Merged

Daniel Meppiel (danielmeppiel) merged 6 commits into
mainfrom
danielmeppiel-ci-release-speed-extract

Conversation

@danielmeppiel

@danielmeppiel Daniel Meppiel (danielmeppiel) commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Named release fix (Lane L freeze exemption, approved by Daniel Meppiel (@danielmeppiel) on 2026-11-17). It extracts the three biggest release-CI wins from #3582 into one PR scoped to build-release.yml and its scripts and tests. Releasing does not change: push a v* tag. What changes is how much work that tag run does, and how fast every other run is.

How to release

# Optional fast path: validate the exact commit you will tag (~35 min, or use last night's run)
gh api repos/microsoft/apm/dispatches -f event_type=manual-build-release
# ...wait for that run to finish green, then:
git tag v0.X.Y <main-sha> && git push origin v0.X.Y
  • If that SHA already has a green full run (the nightly schedule or a manual-build-release dispatch on main), the tag run is promoted. It builds the 5 binaries, smoke-tests them, runs release validation and publishes in about 5 min.
  • Otherwise the tag run validates everything itself, as today, and then publishes. That takes about 35 min instead of about 82.
  • Either way, nothing publishes unless the exact SHA passed full validation, earlier or in the same run. A failed evidence lookup falls back to full validation, never to promotion.

Pipeline modes

flowchart TD
  E{"Trigger"} -->|"push to main"| P["push<br/>unit tests + Linux/Windows binaries<br/>no macOS, no publish"]
  E -->|"nightly schedule or<br/>repository_dispatch"| F["full<br/>every lane, no publish<br/>green run = evidence for its SHA"]
  E -->|"v* tag"| L{"green full run on main<br/>for this exact SHA?<br/>(lookup error = no)"}
  L -->|"yes"| PR["promoted<br/>build + smoke + release validation<br/>then publish (~5 min)"]
  L -->|"no"| FT["full validation in this run<br/>then publish (~35 min)"]
Loading

Full-run topology

Each platform's lanes start as soon as that platform's binary is built. There is no whole-matrix barrier, and macOS still runs exactly two jobs.

flowchart LR
  RP["Release Plan"] --> IP["Integration Shard Plan<br/>(durations: nightly artifact,<br/>else committed seed)"]
  RP --> BX["Build & Test<br/>linux x86_64"]
  RP --> BA["Build & Test<br/>linux arm64"]
  RP --> BW["Build & Test<br/>windows"]
  RP --> MI["Build & Validate<br/>macOS x86_64"]
  RP --> MA["Build & Validate<br/>macOS arm64"]
  IP --> IX
  IP --> IA
  BX --> IX["x86_64 shards 1..N<br/>hermetic + architecture contracts"]
  BX --> NW["network lane<br/>retries failed tests (max 2)"]
  BA --> IA["arm64 shards 1..M<br/>hermetic"]
  BW --> IW["Windows integration"]
  BX --> RV["Release Validation<br/>linux x86_64 / arm64"]
  BA --> RV
  BW --> RVW["Release Validation<br/>windows"]
  IX & IA & NW & IW --> S["Integration Tests Summary<br/>requires every lane,<br/>publishes durations"]
  S & RV & RVW & MI & MA --> CR{"Create GitHub Release<br/>pushed tag only"}
  CR --> PY["PyPI, docs, GH-AW compat"]
Loading

Promoted tag (fast path)

sequenceDiagram
  participant M as Maintainer
  participant GH as GitHub Actions
  participant RP as Release Plan
  M->>GH: repository_dispatch manual-build-release (main @ SHA)
  GH-->>M: full run green (evidence)
  M->>GH: git push origin vX.Y.Z (tag on SHA)
  GH->>RP: release_plan.py (actions: read)
  RP->>GH: list build-release.yml runs for head_sha=SHA
  RP-->>GH: mode=promoted, publish=true
  GH->>GH: build 5 binaries, smoke, release validation
  GH->>GH: Create GitHub Release, PyPI, docs
Loading

What changed

  1. Sharded Linux integration. x86_64 and arm64 integration is split across runners by measured duration. scripts/ci_sharding.py, the seeds in .github/ci-durations/ and the Integration Shard Plan job keep xdist groups and heavy modules whole. Architecture-linter contracts (architecture_contract) run once on x86_64 and share their baselines across workers (tests/utils/xdist_shared.py). Network-dependent tests run in their own lane, which retries only failed tests. On the current corpus the planner produces 7 x86_64 shards and 3 arm64 shards of about 7.5 min of test time each. The nightly run publishes measured durations for the next plan.
  2. Slimmer macOS. Still one job per architecture, and neither job runs on push. arm64 runs the unit suite plus lifecycle_smoke or windows_compat or platform_sensitive. Intel runs windows_compat or platform_sensitive. Seven OS-, filesystem- or shell-sensitive integration modules are now marked platform_sensitive. The platform-agnostic remainder runs once on Linux.
  3. Tag promotion. scripts/release_plan.py picks push, full or promoted. Promotion evidence is only a green schedule or repository_dispatch run of build-release.yml on main for the exact SHA. This is stricter than ci: faster PR, merge-queue and release CI (per-platform fan-out, tag promotion, duration sharding) #3582, which also accepted a main push plus a merge-queue run; main's merge queue covers Linux x86_64 integration only. There is no workflow_dispatch (test_security_workflow_contracts.py).

Builds are now one job per platform (build-linux-x86, build-linux-arm, build-windows) and share their steps through a YAML anchor. Check names are unchanged: Build & Test (...), Build & Validate (macOS ...), Release Validation (...), Create GitHub Release. The only required check, gate, is untouched, and no ruleset requires build-release.yml checks.

Expected timings (from the #3582 rehearsals; post-merge numbers will be posted here):

Path Before After
Full validation (nightly / dispatch / unpromoted tag) ~82 min 31-35 min
Linux x86_64 integration 68.7 min shards of 7-11 min
macOS arm64 73.5 min ~31 min
macOS Intel 36.6 min 9-12 min
Promoted tag ~82 min ~5 min

Issue and approved scope

Issue: Refs #3581

Human scope-approval comment: #3581 (comment), which supersedes #3581 (comment) after the "keep releasing on a plain tag" change. Context: #3581 (comment)

Does this PR complete the issue, or what remains? Partial. The PR-time (ci.yml) and merge-queue (ci-integration.yml, merge-gate.yml) changes stay in #3582, which remains open.

Type of change

  • Bug fix
  • New feature
  • Documentation
  • Maintenance / refactor

Testing

  • Tested locally
  • All existing tests pass
  • Added tests for new functionality (if applicable)

Guardrails:

  • test_lane_selection_partitions_the_non_live_corpus: the lanes partition the non-live corpus, checked by marker algebra.
  • test_macos_capacity_is_two_jobs_with_no_fan_out: at most two macOS jobs.
  • Publication-gate invariants plus new term-removal mutations: create-release needs a pushed tag and every release-bound lane, and only promoted mode may skip integration.
  • test_release_plan.py: strict evidence; push and merge-queue runs never promote; a lookup failure never promotes; an unpromoted tag validates fully and still publishes.
  • test_ci_sharding.py: every node lands in exactly one shard, with xdist groups and modules whole.

Validation (one process, no xdist):

Command Result
pytest tests/unit/scripts/test_release_plan.py tests/unit/scripts/test_ci_sharding.py tests/unit/test_xdist_shared.py -p no:xdist 53 passed (first head)
pytest tests/unit/test_platform_contract_workflow.py tests/unit/test_windows_signing_contract.py tests/unit/test_windows_unit_diagnostics.py tests/unit/test_ado_live_workflow_contract.py tests/unit/test_security_workflow_contracts.py tests/unit/test_windows_compat_gate_workflow.py tests/unit/test_docs_contract_workflow.py tests/unit/test_guardrailing_hero_topology.py tests/unit/test_windows_native_symlink_workflow.py tests/unit/integration/test_integration_wiring_invariants.py tests/unit/scripts/test_lifecycle_smoke_sharding.py -p no:xdist 152 passed
pytest tests/unit/scripts/test_release_plan.py tests/unit/test_platform_contract_workflow.py tests/unit/test_security_workflow_contracts.py -p no:xdist (current head) 92 passed
Real collection: not live vs hermetic + network, and hermetic vs arm64 + architecture_contract 12114 = 12012 + 102; 12012 = 10461 + 1551
ruff check / ruff format --check (0.15.12, CI paths) All checks passed / already formatted
pylint --enable=R0801 ... 10.00/10
bash scripts/lint-auth-signals.sh clean
actionlint .github/workflows/build-release.yml clean

Spec conformance (OpenAPM v0.1)

  • N/A -- this PR does not change OpenAPM-observable behaviour.

Extracts the three release-CI wins from #3582 for #3581:

- Linux x86_64/arm64 integration is split across runners by measured
  duration (scripts/ci_sharding.py, .github/ci-durations seeds,
  integration-plan job), keeping xdist groups and heavy modules whole.
  Architecture contracts run once on x86_64 and share their baselines
  across workers; network-dependent tests get their own retrying lane.
- macOS keeps one job per architecture: arm64 runs units plus
  lifecycle_smoke/windows_compat/platform_sensitive, Intel runs
  windows_compat/platform_sensitive.
- scripts/release_plan.py picks push/full/promoted. A tag promotes only
  when its SHA has a green schedule/repository_dispatch run on main, and
  fails closed otherwise; the override is the repository variable
  APM_RELEASE_FULL_VALIDATION_TAG. create-release requires a pushed tag
  and every release-bound lane; only promoted mode may skip integration.

PR-time and merge-queue changes from #3582 are out of scope.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 17, 2026 15:35
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The network lane can count as promotion evidence while its tests are skipped due to a missing opt-in, and grouped-test timing lookups miss measured durations, potentially selecting too few shards.

2 open issues
What changed in this PR

Restructures APM's release pipeline to reduce validation time through parallel Linux testing, focused macOS coverage, and evidence-based tag promotion. This implements the release-focused portion of #3581; PR and merge-queue changes remain in #3582.

Changes:

  • Adds duration-balanced Linux shards, a separate network lane with bounded retries, and shared architecture-test baselines.
  • Limits macOS validation to platform-sensitive subsets while retaining one job per architecture.
  • Introduces push, full, and promoted release modes, with supporting contracts and documentation.
File Description
tests/​workflow_contracts.py Resolves environment references across workflow scopes.
tests/​utils/​xdist_shared.py Adds locked, session-shared fixture caching.
tests/​unit/​test_xdist_shared.py Covers shared caching, lock timeout, and source hashing.
tests/​unit/​test_windows_unit_diagnostics.py Updates diagnostics contracts for the Windows build job.
tests/​unit/​test_windows_signing_contract.py Retargets signing contracts to the Windows build job.
tests/​unit/​test_platform_contract_workflow.py Checks lane selection, sharding, macOS capacity, and publication gates.
tests/​unit/​test_ado_live_workflow_contract.py Updates live-ADO exclusion checks for the new lanes.
tests/​unit/​scripts/​test_release_plan.py Tests release modes and promotion-evidence requirements.
tests/​unit/​scripts/​test_ci_sharding.py Tests partitioning, duration weighting, planning, and recording.
tests/​integration/​test_unix_installer_path_integration.py Marks installer-path tests as platform-sensitive.
tests/​integration/​test_sparse_cone_symlink_repair.py Marks symlink-repair tests as platform-sensitive.
tests/​integration/​test_self_update_release_lifecycle.py Marks self-update lifecycle tests as platform-sensitive.
tests/​integration/​test_mixed_case_materialization_lifecycle.py Marks case-sensitive materialization tests for platform coverage.
tests/​integration/​test_lockfile_symlink_write_e2e.py Marks lockfile symlink tests as platform-sensitive.
tests/​integration/​test_integrators_hooks_execution.py Marks hook-execution tests as platform-sensitive.
tests/​integration/​test_audit_unrecorded_unicode_lifecycle.py Marks Unicode audit lifecycle tests as platform-sensitive.
tests/​integration/​test_architecture_semantic_rule_mutations.py Shares semantic-rule baselines across workers.
tests/​integration/​test_architecture_owner_rule_mutations.py Shares owner-rule baselines across workers.
tests/​integration/​test_architecture_install_compound_mutations.py Removes forced single-worker grouping.
tests/​integration/​test_architecture_contract_compound_mutations.py Removes forced single-worker grouping.
tests/​integration/​test_architecture_confirmed_bypass_mutations.py Removes forced single-worker grouping.
tests/​integration/​test_architecture_compile_inventory_mutations.py Removes forced single-worker grouping.
tests/​integration/​test_architecture_bundle_format.py Removes forced single-worker grouping.
tests/​integration/​conftest.py Classifies architecture contracts before marker selection.
scripts/​test-integration.sh Adds failed-test retries and module-based pytest invocation.
scripts/​release_plan.py Selects release modes and verifies promotion evidence.
scripts/​ci-shard-durations.sh Selects nightly duration data with committed-seed fallback.
scripts/​ci_sharding.py Implements duration-aware shard planning and recording.
pyproject.toml Registers architecture and platform-sensitive markers.
docs/​src/​content/​docs/​contributing/​integration-testing.md Documents release modes and platform test selections.
CHANGELOG.md Records the release-CI changes.
.github/​workflows/​build-release.yml Wires platform builds, integration lanes, and promotion gates.
.github/​instructions/​cicd.instructions.md Updates contributor guidance for the release topology.
.apm/​instructions/​cicd.instructions.md Updates the corresponding APM instruction source.

Review effort: Balanced

Comment thread .github/workflows/build-release.yml Resolved
Comment thread scripts/ci_sharding.py Resolved
A v* tag without promotion evidence now runs full validation in the tag
run and then publishes, instead of failing at Release Plan. A tag whose
SHA has a green schedule/repository_dispatch run on main is still
promoted without re-running the suites. A failed evidence lookup falls
back to full validation. The APM_RELEASE_FULL_VALIDATION_TAG override is
no longer needed and is removed.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot removed the panel-review Request an advisory PR review; consumed after review. Not human approval or a merge gate. label Nov 17, 2026
@github-actions

Copy link
Copy Markdown

Advisory review: PR #3771

Release CI pipeline sharded and gated by tag-promotion script, cutting wall-clock time while preserving fail-closed validation.

panel-mode=lean; personas=supply-chain-security-expert

Summary

The fail-closed design is sound: a tag publishes only when a green schedule or repository_dispatch run of build-release.yml exists on main for the exact SHA. The override variable names one tag. The panel found 0 blocking, 0 recommended and 3 nit findings. Only the supply-chain lens ran, and its findings come from the PR text and scripts/release_plan.py, not a full diff read. Architecture, test-coverage and DevX personas were not run in this lean pass.

Persona Blocking Recommended Nit Summary
supply-chain-security-expert 0 0 3 Promotion gate is fail-closed; hardening opportunities only

Recommended follow-ups (curated)

  1. Staleness cap on evidence runs (highest signal). find_evidence accepts a green run of any age for the same SHA. Consider a time bound.
  2. Document the 100-run limit. per_page=100 with no pagination fails closed. A one-line comment prevents a well-meant "fix" that weakens the gate.
  3. Job-level conclusions. Run-level success could mask a future continue-on-error on a validation job. Add a comment or guard.

Ship recommendation

ship_with_followups. There are no blocking findings, and the PR body reports passing tests. Track the follow-ups above in a small post-merge PR. CODEOWNERS review requests (@sergio-sisternes-epam) remain authoritative.

cc Daniel Meppiel (@danielmeppiel) @sergio-sisternes-epam

Generated by PR Review Panel for #3771 · copilot · auto · 172.7 AIC · ⌖ 0.539 AIC · ⊞ 5.8K · ◷

…summary

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- The network lane sets APM_RUN_INTEGRATION_TESTS=1, so its
  requires_network_integration tests run instead of skipping.
- ci_sharding resolves a grouped test's duration with or without the
  @<group> suffix that xdist loadgroup adds, so the planner (plain
  collection) and the shards agree on measured weights.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@danielmeppiel

Copy link
Copy Markdown
Collaborator Author

APM Review Panel: ship_with_followups

Shards Linux integration into duration-balanced lanes, slims macOS to targeted markers, and replaces the tag-override gate with evidence-based promotion. Full-validation wall time drops from about 82 min to about 35 min, and promoted-tag runs to about 5 min.

panel-mode=lean; personas=supply-chain-security-expert,test-coverage-expert,python-architect,apm-ceo

cc Sergio Sisternes (@sergio-sisternes-epam) -- a fresh advisory pass is ready for your review.

No panelist raised a blocking finding, and none disagreed on severity. The python-architect confirms that release-plan remains the single authority for the run mode, and that the tests exercise every tag path: promoted, full validation in the tag run, and lookup failure falling back to full validation. The test-coverage-expert's two recommended findings are both folded on this head with new mutation tests: an unpinned every-lane check in Integration Tests Summary, and an unasserted no-evidence step-summary line. The Copilot reviewer's two threads are fixed and resolved: the network lane opt-in, and grouped-duration lookup.

The supply-chain-security-expert raised one open point. Because a tag without evidence now validates fully and publishes instead of being refused, the pipeline no longer enforces that the tagged commit is on main. Two facts bound the risk. The releases tag ruleset already limits v* tag creation to a bypass role, which is narrower than push rights on main. The pre-PR pipeline also published any tag after full validation, so this restores earlier behavior rather than adding a regression. A main-ancestry guard is worthwhile hardening, but it does not need to block a CI change that must land before the v0.36.0 tag.

Aligned with: secure by default (the publication gate is unchanged, and no tag ships without a green full validation of its SHA); governed by policy (one decision point drives every downstream job, with no manual override left); multi-harness, multi-host (duration-balanced shards and the platform_sensitive marker algebra cover every target without duplicate work).

Panel summary

Persona B R N Takeaway
Supply Chain Security Expert 0 1 0 Publication integrity is sound. Main-branch provenance for tags without evidence now rests on the tag ruleset.
Test Coverage Expert 0 2 0 Strong mutation-style coverage. Both gaps are folded on this head.
Python Architect 0 0 0 Single-owner mode authority is preserved, and the topology is correct for every tag path.

B = blocking-severity findings, R = recommended, N = nits.
Counts are signal strength, not gates. The maintainer ships.

Top 1 follow-ups

  1. [Supply Chain Security Expert] Add a main-ancestry check (compare API or merge-base) for tags, so that only commits reachable from main can publish. -- This restores the provenance invariant inside the pipeline instead of relying only on the tag ruleset.

Recommendation

Merge now to unblock the v0.36.0 tag. The main-ancestry guard is hardening, not a regression fix; track it as an issue before the next release cycle. Every other panelist finding is resolved on this head.


Full per-persona findings

Supply Chain Security Expert

  • [recommended] The no-evidence tag path removes the main-branch provenance invariant from the release pipeline, at scripts/release_plan.py:79
    Any v* tag without evidence now gets full validation and is published. The tested binary keeps its integrity, but "released code went through main's review" now depends on the tag ruleset.
    Suggested: Check ancestry with git merge-base --is-ancestor or the compare API, or pin the tag ruleset in a contract test.

Test Coverage Expert

  • [recommended] Nothing pinned the "Require every integration lane" step in Integration Tests Summary. Folded: _assert_integration_summary_requires_every_lane plus four mutation tests.
  • [recommended] The "Evidence: none:" step-summary diagnostic was not asserted on the no-evidence main() path. Folded: test_main_writes_outputs_and_summary now asserts it.

Python Architect

No findings.

This panel is advisory. It does not block merge. Re-apply the
panel-review label after addressing feedback to re-run.


Generated by autopilot-pr-review-worker. This comment is AI-generated and may contain errors.

@danielmeppiel
Daniel Meppiel (danielmeppiel) merged commit 3133a7f into main Nov 17, 2026
26 checks passed
@danielmeppiel
Daniel Meppiel (danielmeppiel) deleted the danielmeppiel-ci-release-speed-extract branch November 17, 2026 16:12
Daniel Meppiel (danielmeppiel) added a commit that referenced this pull request Nov 17, 2026
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Daniel Meppiel (danielmeppiel) added a commit that referenced this pull request Nov 17, 2026
Resolves the CHANGELOG [Unreleased] conflict with #3771: keep both
entries, Changed before Fixed per Keep a Changelog.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Daniel Meppiel (danielmeppiel) added a commit that referenced this pull request Nov 18, 2026
* chore: release v0.36.1

Bump apm-cli to 0.36.1 in pyproject.toml and the apm-cli entry of uv.lock (no uv lock re-resolve), and cut the [Unreleased] CHANGELOG block into [0.36.1] - 2026-11-18 with upgrade notes. The #3771 and #3776 notes move into [0.36.0], where they shipped. Lint mirror (ruff check, ruff format --check, pylint R0801, auth-signals) is clean on main 70554b6.

Post-merge: run the pre-tag gate on the merge commit, then tag v0.36.1.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* docs(changelog): do not claim ADO issues fixed before a live-org check

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* docs(changelog): close the Azure DevOps issues in the 0.36.1 entry

Reverts the previous wording change at @danielmeppiel's direction: #3752, #3753 and #3758 are closed by #3790.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* docs(changelog): note SSE removals and the COPILOT_HOME skill scope in 0.36.1 upgrade notes

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* docs(changelog): drop the 0.36.1 upgrade-notes paragraph

Patch releases carry upgrade details in their entries, as before.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: danielmeppiel <danielmeppiel@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Daniel Meppiel (danielmeppiel) added a commit that referenced this pull request Nov 19, 2026
* docs(changelog): make 0.36.0 onwards legible and concise

Rewrites [Unreleased], [0.36.1] and [0.36.0] as one short line per user-visible change, with Highlights and a Breaking changes and upgrade notes list per version, grouped PRs and internal-only entries dropped (#3637, #3771, #3794, #3796, #3877). Credits, PR references, the 0.36.1 issue closures and the live Azure DevOps caveat are kept. Sections from 0.35.1 down are unchanged. The changelog instructions now describe this style.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* chore(lock): record the new changelog instructions hash

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* docs: note the shared .mcp.json and VS Code Local mode in 0.37 upgrade notes

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* docs(changelog): restore accuracy qualifiers flagged in review

Restores the static env caveat for .mcp.json, the pack default-format
qualifier, Copilot CLI scope for COPILOT_HOME, the 0.36.1 frozen-lockfile
recovery step, and the bundle install --target qualifier.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: danielmeppiel <danielmeppiel@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants